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Introduction 



Achievement gaps — that is, differences in academic achievement among ethnic and 
socioeconomic groups — have long been an issue of concern in education. The No Child Left 
Behind Act of 2001 (NCLB) began, upon its institution in 2002, to hold states more directly 
accountable for the progress of all students, including students of color and students of low 
socioeconomic status (SES). These groups of students have historically performed less well 
than their White or upper-class peers. 

In 2001, prior to the passage of NCLB, officials from Kanawha County Schools (KCS) 
and local community members began addressing educational inequities for African American 
students. The West Virginia school district had been the focus of several negative media reports 
and had been under scrutiny for ethnicity-related issues including the social promotion of 
African American students and inadequate performance on state assessments at schools serving 
high percentages of African American students. The district had not been successful in its 
attempts to help African American students make gains in achievement test and ACT scores, 
college preparedness, or high school graduation rates. Further, KCS had received a citation 
from the U.S. Office of Civil Rights because of the disproportional representation of African 
American students in certain special education categories. These factors prompted KCS 
officials to see a need to reach out to local African American community leaders and take steps 
to address the needs of African American students served by the district. 

In May of 2001, KCS officials approached Appalachia Educational Laboratory (AEL) 
at Edvantia, Inc. (hereafter, “the lab” or “laboratory”) staff members and requested assistance 
in establishing a dialogue with African American community leaders about the needs and 
achievement of African American children in the district. The lab, which had begun a 
partnership with faith-based organizations in the local community to expand educational 
support for African American and low-income youth (see Keyes & Kusimo, 2004, for more 
information), agreed to facilitate the process. As a result, the school district and community 
leaders began a 16-month dialogue and collaboration that focused on race and education. The 
primary function of the initiative was to increase understanding of the educational experiences 
and perceptions of African American students and of school district personnel and thereby 
improve the quality of education for African American students in the district. The community 
members of the initiative adopted the name Maximizing the Achievement of African American 
Children in Kanawha (MAACK). Although the formal dialogue process ended in 2002, the 
MAACK Community Initiative continues to function and to collaborate with KCS to find ways 
to decrease achievement gaps between African American students and their White peers. 

Community leaders from the MAACK work to mobilize the broader community to take 
action and become involved in efforts to improve the academic achievement of African 
American and low-income students. Since the initiative’s inception, Edvantia has supported the 
initiative through assistance with record keeping, mailings, printing, and copying. Edvantia 
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staff also have provided technological support for the initiative’s Web site 
(www.maacknews.org), funded part-time community coordinators, and helped the group plan 
and publicize its activities. Edvantia staff undertook these support activities to help the 
initiative (1) increase communication and cooperation between the schools that serve African 
American and low-income students and their communities, and (2) increase community 
members’ efforts to support, advocate for, and monitor the quality of education. 

During the fall of 2005, evaluation data were collected and analyzed to determine the 
nature of Edvantia’ s lab-supported involvement, and to examine the extent to which delivered 
services were useful for the MAACK Community Initiative. A primary audience for this report 
is project staff, who served as the key facilitators. 



Methods 



Participants 

The participants in this evaluation were selected purposefully. They were laboratory 
staff members who were involved in MAACK activities, and they were educators and/or 
community members who served in leadership roles in the MAACK. Ultimately, two staff 
members, one consultant, one school district administrator, and three community members 
(including two co-chairs of the MAACK initiative) participated in this study. 

With the exception of one laboratory staff member who began working with the 
MAACK in 2004, all participants had been active in the MAACK project for at least three 
years, and four of the participants had been MAACK members since its inception. In addition 
to representing long-term experience with the MAACK, the sample was also racially diverse: 
three participants were White, and four were African American. 



Design and Data Collection 

The intent of this study was to describe the perceptions and experiences of a particular 
group (MAACK Community Initiative leaders and participants). A qualitative research 
paradigm was selected for this study because it was largely an investigation of a social 
phenomenon, undertaken by intensively comparing, contrasting, and classifying participants’ 
experiences, and perceptions (Yin, 2003). 

The research strategy developed for this study involved a series of in-depth interviews 
with MAACK community members, the school administrator, and Edvantia staff and 
consultant who worked with the MAACK. This qualitative methodology corresponded to the 
nature of the data sought: attitudes, self-reported behavior, and exploration of interactions and 
exchanges (LeCompte & Preissle, 1993; Yin, 2003). The evaluator (an external consultant) 
developed a semi-structured interview protocol (see Appendix A for protocol and Appendix B 
for consent information), and all interviews were conducted in person at Edvantia’ s offices in 
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Charleston, West Virginia, during late October 2005. Interviews were recorded on audiotape, 
and the recordings and evaluator’s notes served as the data for analysis. 



Data Analysis 

Interview transcripts were coded and summarized according to general descriptive 
categories. Pattern coding (Fetterman, 1989; Yin, 2003) was used to discover patterns among 
individuals and descriptive categories. Before making any assertions regarding a pattern, the 
evaluator conducted a search for data that opposed or were inconsistent with these conclusions. 
The portrait of the interviewees’ experiences is derived from this work. 



Findings 



The results of the interviews are presented here. The following section describes the 
role of the regional educational laboratory at Edvantia in the MAACK Community Initiative, 
leadership in school-community relationships, and efforts to facilitate community collaboration 
and mobilization. 



Role in the MAACK Community Initiative 

Laboratory staff members who were active with the MAACK were perceived by all 
non-staff interviewees as providers of support, facilitators, organizers, brainstormers, and, most 
significantly, leaders. Interestingly, none of the Edvantia staff perceived themselves as leaders 
in the MAACK. In particular, one explained, “We are here to provide support and research. 

The MAACK is led by community members and their cochairs.” 



School-Community Relationships Leadership 

Laboratory staff members were to provide support and facilitation for school- 
community forums in which KCS officials and African American community members in 
Kanawha County learned how to identify, discuss, and address issues affecting the education of 
African American and low-SES students. What follows is a description of how the lab staff and 
key MAACK members carried out these goals. 

Every Thursday, the MAACK Community Initiative met at a church in an African 
American neighborhood. According to the KCS and community participants, the laboratory 
staff emerged as the leaders of these meetings, despite the presence of MAACK committee 
cochairs. This perception is based on the lab’s role in organizing the meetings, which included 
such activities as arranging for a meeting space, distributing information about when and where 
the meetings would occur, setting the agenda for the meetings, and facilitating the discussion at 
the meetings. 
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Interviews revealed that laboratory staff carried out the following activities related to 
leadership in school-community relationships: 



• Laboratory staff facilitated learning and discussion about issues affecting the education 
of African American and low-SES students. For example, one interviewee (a school 
official) reported that there was significant tension between the African American 
community members present at the meetings and the educators; in an attempt to break 
down the barriers between the two groups, lab staff members engaged the MAACK in 
role-playing activities. According to the school official, “As a result of activities like 
this, the community was able to get beyond ‘us-versus-them’ thinking and begin to 
discuss problems affecting African American students and solutions to those problems.” 

• Laboratory staff worked with the community to interpret and disseminate information 
about district graduation rates, test results, and course enrollment patterns disaggregated 
by ethnicity and SES. All of the MAACK members interviewed for this study suggested 
that sharing data about African American students was one of the most important 
aspects of the lab's role in the MAACK and may have been the most important. One 
MAACK member explained that “sharing achievement data with educators, counselors, 
and community members helped to solidify the [reality of] problems with the 
achievement gap in KCS. While it was emotionally difficult for everyone to see the 
data, we all recognized how profound the problem is: We are not educating our Black 
kids.” 

• Laboratory staff worked with school leaders and African American community 
representatives in Kanawha County. According to interviewees — laboratory staff 
members, in particular — occasionally, the MAACK invited schoolteachers, counselors, 
principals, and local ministers from predominantly Black churches to MAACK 
meetings. These meetings seemed to gamer the largest attendance of African American 
community members because, as one MAACK member explained, “They provided a 
venue for parents to ask questions about the schools, a place where all of the community 
members could get honest feedback. The result was that communication between the 
schools and parents increased, and church leaders began to promote the MAACK at 
their churches.” In addition to generating discussion within a broader community, 
laboratory staff also taught the community leaders how to read data and also trained 
some community leaders in how to use a variety of reading tools intended to help the 
community close the achievement gap. 

Although these activities, together, seemed to engage the schools and the African 
American community in a meaningful dialogue, the MAACK members reported significant 
challenges in carrying out these tasks. One member thought that collecting and disseminating 
the data (in a timely manner) were difficult tasks, but well worth the effort. Additionally, three 
members thought that facilitating discussion around the data was taxing because the 
community’s response was emotionally charged. However, the biggest challenge, according to 
all interviewees, was “getting the parents to come to the meetings.” One member explained that 
she would transport some parents herself from time to time; nevertheless, attendance at the 
meetings waxed and waned. For instance, at one meeting there were at least 100 people present, 
but most often there were fewer than 20. 
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Role in Collaboration and Mobilization 

As mentioned previously, laboratory staff planned to collaborate with African American 
community leaders and others in Kanawha County to develop materials and strategies useful in 
community mobilization efforts to close the achievement gap. However, although most of the 
interviewees believed there was some collaboration, few could point to specific examples of 
such activities. Ultimately, the collaboration was perceived to focus on the following activities: 



• Laboratory staff developed online and print materials to inform community members 
about issues affecting and resources supporting student learning, gathered research 
materials around academic achievement, and, with the help of the MAACK steering 
committee, produced brochures for parents. These materials focused on three school 
levels: elementary school, middle school, and high school. The brochures provided 
information about what students should be doing/learning at different grade levels. Staff 
also helped the MAACK develop a Web site that provided similar but more extensive 
information. 

• Laboratory staff worked with the school district to train community academic advisors. 
As mentioned earlier, staff also helped train community members to serve as 
community academic advisors. These academic advisors helped answer questions and 
provided guidance to both teachers and parents of African American students. 

• As part of a separate but related project, laboratory staff helped the schools develop 
culturally sensitive curriculum materials and trained educators to implement them in the 
schools (described in Hughes et al., 2004). According to one MAACK member, “They 
not only helped us create it, but they also followed through and carried out observations 
as we implemented it in our classrooms. This collaboration was indispensable.” 

While the MAACK members were proud of their collaboration with the laboratory, they 
recognized that there were some daunting challenges in such a venture. Specifically, one 
member expressed disappointment with the MAACK’ s attempt to engage the African 
American ministers in the project. She felt that “early on, the ministers all showed up to the 
meetings and implied that they would be willing to collaborate with the MAACK. 
Unfortunately, as time went by, fewer and fewer ministers participated.” In addition, one 
laboratory staff member perceived tension related to the lab’s role: “.. . staff . . . had to decide 
whether we were advocates or support. If we were advocates, we could actively engage in 
collaborations with community members to help close the achievement gap, but if we were 
support, our ability to collaborate was limited in that regard. In the end, I think we leaned more 
toward support, which meant that our role was simply to provide structure to the organization.” 

Finally, two of the three MAACK members who were interviewed felt that the 
collaboration was missing one key element: capacity building. Interviewees explained that the 
lab provided outstanding leadership and guidance; however, it did not help the MAACK 
develop a strategy for continuing the work of the organization without lab support. As one 
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member explained, “We will continue this work, but we don’t know how. We don't really 
know what we need [in terms of resources] to carry on.” 



Conclusions and Discussion 



The data presented here paint a complex picture of the laboratory’s role in the MAACK 
community initiative. However, some conclusions based on the data can be stated clearly: 



• Laboratory staff provided necessary structure and organization to MAACK meetings 
and activities. 

• Laboratory staff gathered and disseminated educational achievement data and other 
relevant school data. 

• Laboratory staff facilitated challenging and emotionally charged discussions between 
community members and school district personnel regarding the achievement gap. 

• Laboratory staff collaborated with MAACK members to produce relevant print 
documents and online materials to educate and engage African American parents and 
students, but most people interviewed could not describe particular instances of 
collaboration. 

The involvement and support of lab staff with the MAACK Community Initiative seemed to 
include activities that could advance the goals of (1) increasing communication and cooperation 
between the schools that serve African American and low-income students and their 
communities, and (2) increasing community members’ efforts to support, advocate for, and 
monitor the quality of education. However, a few points are worth further consideration. 

Although the laboratory staff involved with the MAACK project did not perceive 
themselves as leaders of the community initiative, all of the non-staff interviewees did hold that 
perception. In fact, most of them believed that the lab's leadership was the key factor in 
mobilizing the meetings and carrying out various activities. Consequently, the MAACK 
members expressed anxiety about the future of the MAACK without continuing guidance and 
support from the lab. Perhaps these perceptions contributed to the MAACK members’ thoughts 
on collaboration. They seem to have sensed that laboratory staff were more like leaders than 
collaborators in the MAACK activities. 

The inconsistency between the perceptions of MAACK community members and 
laboratory staff regarding leadership of the initiative is interesting. Staff members thought they 
communicated to community members that the MAACK belonged to the community and that 
the lab was to provide support only. However, as evidenced by the differences in perceptions 
and by the anxiety of community members about the laboratory’s imminent withdrawal from 
involvement, the community members may not have understood fully or accurately the 
distinction between support and leadership. Likewise, staff may not have understood the 
positions or perceptions of community members. The perception among community members 
that laboratory staff were leaders in the initiative could be due to the structural support 
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provided by the lab. The community members may have associated such support with 
leadership, regardless of the actions of staff members or any statements to the contrary. Further, 
the facilitation of meetings by lab staff may have been perceived as leadership. If lab staff were 
not conscious of community members’ perceptions, they would not have recognized the need to 
taken action to further empower community members to take over facilitation responsibilities. 
This lack of awareness on the part of laboratory staff is evident in the perceptual differences. 

In the present, it may be important for staff to work with MAACK Community 
Initiative leaders to develop an exit strategy and to help build capacity within the initiative to 
ensure that the work will continue without continuing laboratory support and guidance. In 
future efforts of this nature, it may be important to develop such an exit strategy at the outset of 
the collaboration. Further, processes should instituted to ensure that staff are aware of the 
perceptions and beliefs of group members and to ensure that group members and laboratory 
staff have shared understandings of definitions, roles, and responsibilities. Explicit steps to 
ensure capacity building and empowerment of the community group or initiative throughout the 
partnership should also be planned and instituted at the outset. Such strategies may include 
encouraging tangible, in-kind contributions from group members (e.g., membership dues, 
contributions of supplies). These contributions may foster the perception among community 
members that they “own” the group and are equally invested in its success. 
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Appendix A 

Protocol for Evaluation Interviews 




MAACK Project Interview Protocol 



The goal of this protocol is to gather information from community members, school 
district officials, and Edvantia staff members who were involved in the MAACK project in an 
effort to describe their role in MAACK project activities and implementation. Thus, the 
questions are driven by the MAACK project objectives: (1) to increase communication and 
cooperation between the schools that serve African American and low-income students and 
their communities; and (2) to increase community members’ efforts to support, advocate for, 
and monitor the quality of education. 

Interview Questions 

Leadership in school-community relationships 

Edvantia/ AEL was to provide leadership in school-community forums in which staff of 
Kanawha County (WV) Schools and African American community members in Kanawha 
County learned how to identify, discuss, and address issues impacting the education of African 
American and low-SES students. Did this occur as planned? 

1 . Who were the AEL staff members selected to provide that leadership, and what were 
their roles? 

2. What was your role in the leadership of the school-community forums (if any)? 

3. What were the major leadership activities that you carried out (or those carried out by 
the Edvantia/ AEL staff) related to the school-community forums? 

Probe: 

a. (How) did they/you facilitate learning and discussion about issues impacting the 
education of African American and low-SES students? 

b. (How) did they/you work with the community to interpret and disseminate 
information about district graduation rates, test results, and course enrollment 
patterns disaggregated by ethnicity and SES? 

c. (How) did they/you work with school leaders and African American community 
representatives in Kanawha County, WV, to convene school and community 
leaders? 

d. (How) did they/you disseminate research on the achievement gap and best 
practices for improving minority student achievement with community, school 
leadership and teachers? 

4. How often and where did the forums convene? 




5. What were some of the challenges/benefits associated with the leadership of the 
forums? 



Probe: 



a. (How) did you/they overcome those challenges? 



Collaboration and mobilization 

Edvantia/AEL planned to collaborate with African American community leaders and others in 
Kanawha County, West Virginia, to develop materials and strategies useful in community 
mobilization efforts to close the achievement gap. Did this occur? 

1. Who were the Edvantia/AEL staff members that collaborated with the African- 
American community leaders? 

2. What did you/the Edvantia/AEL members do to facilitate the collaboration? 

Probe: 



a. Develop online and print materials to inform community members about issues 
affecting and resources supporting student learning. 

b. Work with school district to train community academic advisors. 

c. Provide staff support for community leaders as they plan efforts to mobilize 
community members to address the achievement gap issue. 

d. Work with community members and leaders to implement community 
mobilization plans. 



3. What were some of the challenges/benefits associated with the collaboration efforts? 
Probe: 

a. (How) did you/they overcome those challenges? 



4. Were there any MAACK project activities, other than those originally planned for 
Edvantia/AEL staff members, that you participated in? If so please describe them. 
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Informed Consent Information for Evaluation Interviews 




DVANTIA 



MAACK Community Initiative (School- 
Community Connections) 
Evaluation Interviews 



Contacts: 

Georgia Hughes, Research & Evaluation Specialist I (304-347-0413; georgia.hughes@edvantia.org) 



Edvantia, Inc. (formerly AEL) in Charleston has collaborated with the MAACK Community 
Initiative to support and facilitate the group’s activities and provide guidance as needed. The 
company is now evaluating its efforts to determine whether or not the support provided was 
useful or beneficial to the MAACK. As part of the evaluation, we are conducting interviews 
with knowledgeable people such as yourself to find out more about Edvantia’ s work. 

We are asking that you give us your honest feedback and opinions about various aspects of 
Edvantia’ s support of the MAACK. The interview should require about 30 minutes of your 
time. Participating in this interview should involve no risks to you that are any greater than 
those you experience in your daily life. Your participation in this survey is voluntary; you may 
discontinue your involvement at any time without any reprisal or penalty. 

Edvantia staff will take every reasonable precaution to protect your confidentiality throughout 
this project 1 . The information that you provide will be combined with information from other 
interviewees and reported all together. Your individual responses will not be singled out and 
you will never be identified by name or role in any report(s). Only your interviewer (Aaron 
Baker) and Georgia Hughes will have access to your responses. 

Taking part in this interview will allow you to provide important and useful feedback to 
Edvantia, which will use the results of this research to inform future efforts with similar 
initiatives. Although you will not be compensated directly for your participation, your thoughts 
and suggestions will help Edvantia make important decisions about how they can best serve the 
needs of educators and community members. 

Your participation in the interview indicates your consent. If you have any questions or 
concerns about this interview, including its puipose or expectations of you as a participant, 
please contact Georgia Hughes (304-347-0413 or georgia.hughes@edvantia.org). If you have 
any questions or concerns about your rights as a participant in this research, you may contact 
Dr. Merrill Meehan, Edvantia’ s Institutional Review Board (IRB) Chair (304-347-0432 or 
merrill . meehan @ edvantia. org ) . 



1 Data collected for research purposes are stored in compliance with ISO 17799 requirements for access, security, 
and redundancy. Data are stored in an encrypted format in a centralized, electronically and physically secure 
server at Edvantia for a period not to exceed five years. All electronic data of a personal nature are safeguarded 
and available only to those project leaders, staff, and technologists having a need to know within the specific 
criteria as set forth in the approved project plan. The Edvantia Institutional Review Board (IRB) has the authority 
to inspect consent records and data files only to assure compliance with approved procedures. 




Appendix C 



Evaluation Standards Checklist 




Checklist for Applying the Program Evaluation Standards 



To interpret the information provided on this form, the reader needs to refer to the full text of the standards as they appear 
in Joint Committee on Standards for Educational Evaluation, The Program Evaluation Standards (1994), Thousand Oaks, 
CA, Sage. 



The Standards were consulted and used as indicated in the table below (check as appropriate): 

_ , i o i — , ■ The Standard The Standard was The Standard was Th< 

Standard & Descriptor ,, , . „ ,, , ,, , 

was addressed partially addressed not addressed n 



The Standard was The Standard was 
not addressed not applicable 



/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ Consultant hired to 
conduct evaluation 



Stakeholder Identification 

Evaluation Credibility 
Information Scope and Selection 
Values Identification 
Report Clarity 

Report Timeliness and Dissemination 

Evaluation Impact 

Practical Procedures 

Political Viability 

Cost Effectiveness 

Service Orientation 

Formal Agreements 

Rights of Human Subjects 

Human Interactions 

Complete and Fair Assessment 

Disclosure of Findings 

Conflict of Interest 

Fiscal Responsibility 
Program Documentation 
Context Analysis 

Described Purposes and Procedures 
Defensible Information Sources 
Valid Information 
Reliable Information 
Systematic Information 
Analysis of Quantitative Information 
Analysis of Qualitative Information 
Justified Conclusions 
Impartial Reporting 

Metaevaluation 



The Program Evaluation Standards (1994, Sage) guided the development of this (check one): 

□ Request for evaluation plan/design/proposal 0 Evaluation report 

□ Evaluation plan/design/proposal D Other: 

□ Evaluation contract 



y May be metaevaluated 
at a later time 



Name: 



Georgia K. Hughes 

(typed) 



Date: November 3. 2005 



Position or Title: 

Agency: 

Address: 



(signature) 

Research & Evaluation Specialist II 

Edvantia, Inc, (formerly AEL). Institute for the Advancement of Research in Education 

P.O. Box 1348 

Charleston. WV 25325-1348 



Relationship to Document: 



Second author 

(e.g., author of document, evaluation team leader, external auditor, internal auditor) 









